Introduction 


The Learning Assessment in Missouri Postsecondary Education (LAMP) Advisory Council was 
created to consider the issues surrounding statewide learning assessment and to make 
recommendations for policy. LAMP is comprised of a voluntary group of assessment 
professionals, postsecondary faculty and administrators, secondary educators and administrators, 
and Missouri Department of Higher Education (MDHE) staff. 


The work of the Advisory Council was completed within subcommittees: Assessment Practices 
Subcommittee, Communications/Next Steps Subcommittee, and the Literature Review 
Subcommittee. This report reflects the progress-to-date of the LAMP Subcommittees; their draft 
documents are included in this report. 
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LAMP Charge 


Coordinating Board for Higher Education 
Learning Assessment in Missouri Postsecondary Education (LAMP) 


Higher education institutions must demonstrate good stewardship of both the resources and 
students with which they have been entrusted. Legislators and the public want quick, easy-to- 
understand information to ensure that postsecondary institutions are indeed held responsible for 
achieving their missions. 

Comprehensive student learning assessment should foster student learning, establish a 
foundation for a culture of continuous improvement, and provide ways to demonstrate 
accountability. These practices provide opportunities for feedback, evaluation, and enhancement 
of instruction and curriculum development for postsecondary administrators and educators. 


The challenge for higher education in Missouri is to create a statewide assessment policy that is 
built upon the foundations of previous statewide efforts [e.g., Missouri Assessment Consortium 
(MAC), Missouri Developmental Education Consortium (MoDEC), and Missouri Consortium 
for Measuring Value-Added Student Learning (MVASL)] and driven by the improvement of 
student learning while responding to the call for accountability. A cohesive statewide 
assessment approach must develop a greater understanding of the scope and magnitude of 
assessment in Missouri; gather information on best practices, both local and national; agree upon 
meaningful methods and outcomes; and make appropriate policy recommendations. 


Consensus on student learning assessment issues will support multiple state-level priorities and 
address accompanying areas of policy impact, including the Coordinating Board for Higher 
Education (CBHE) coordinated plan, Imperatives for Change, and the SB 389-mandated 
Curriculum Alignment Initiative. In order to fulfill these mandates, the Commissioner of Higher 
Education, through the authority of the CBHE, has established Learning Assessment in Missouri 
Postsecondary Education (LAMP). LAMP is a voluntary group intentionally composed of a 
cross section of educators and administrators, including MAC members, institutional researchers, 
content area specialists, faculty, administrators, K-12 educators, and assessment specialists. Such 
a dynamic group provides opportunity for collective knowledge development and individual self- 
evaluation of assessment practices. 


The following duties are necessary to carry out this charge: 


1. Perform a review of Missouri postsecondary assessments currently in use 
Perform a review of literature and professional knowledge regarding effective use of 
assessment of student learning for continuous improvement and for accountability 
3. Deliver a report to the Commissioner of Higher Education by June 1, 2009, including: 
a. Summary and analysis of current Missouri practices 
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Review of relevant assessment research 

Policy recommendations 

Impact on existing CBHE policies 

e. Possible pilot projects as proof of concept 
4. Develop and implement a communication plan to publicize, allow feedback, and build 

support at the secondary and postsecondary levels concerning the development of a 
statewide assessment agenda. 

All meetings will be advertised and open to the public. 


aos 
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LAMP Values Document 


Following LAMP’s inaugural meeting in October 2008, it was decided that a smaller group, 
called Next Steps, would be formed to further clarify a direction for the group. The Next Steps 
group developed a draft Principles/Values of Inclusion document that outline a set of values and 
principles that LAMP will use to guide its work. The principles are not meant to be principles of 
assessment, but rather, are meant to form the foundational principles on which LAMP’s work 
will proceed. 


LEARNING ASSESSMENT IN MISSOURI POSTSECONDARY 
EDUCATION (LAMP) 
PRINCIPLES FOR INCLUSION/VALUES DOCUMENT 


These values have been developed and refined by the "Next Steps" Group formed to clarify goals 
and direction after the first LAMP meeting on October 23, 2008. Note that these principles are 
not meant as principles of assessment, but are meant to form the foundational principles on 
which LAMP's work will proceed. 


1. We want all institutions and sectors (e.g. secondary, postsecondary public, private, two- 
year, four-year) to be engaged in the process of making LAMP policy recommendations 
to the Commissioner 

2. We acknowledge the opportunity for assessment to support accountability to our various 
publics 

3. We value the mission and autonomy of each postsecondary institution 

4. The primary purposes of assessment are to improve student learning, enhance curriculum 
development and instructional delivery, and support institutional continuous 
improvement 

5. We recognize the existing assessment work in the state and will seek to build on that 
foundation where possible 

6. We strive to reduce duplication of effort in assessment 

7. We seek a collegial process for sharing assessments, best practices, and benchmarking for 
improvement 

8. We focus on aspirational goals 

9. We value using assessments to accomplish seamless educational transitions 

10. We acknowledge the responsibility for assessment to respond to existing legislation 


6|Page 


LAMP Policy Guidance Document 


The MDHE staff provided the LAMP Advisory Council with the Guidance Document to outline 
the MDHE issues and policy questions that are foundational to the creation of LAMP. These 
issues provide the lens through which LAMP’s tasks are identified. 


LAMP Policy Guidance 


The items below are meant to outline MDHE issues and policy questions that were foundational 
to the creation of LAMP. These are the issues upon which MDHE seeks input and information in 
the form of the LAMP report to the Commissioner for Higher Education. These issues are meant 
to serve as a lens through which LAMPs tasks and goals are to be chosen. Note that the policy 
priorities to address first are listed under number 2, items a, b, and c; the remaining policy issues 
will need to be addressed, but the department recognizes that it is necessary to identify priorities 
for directing resources. 


1. Articulation of Guiding Principles for a statewide Post-secondary Assessment Policy 
a. Building upon prior statewide collaborative work on assessment (Missouri Assessment 
Consortium's "Guiding Principles on Assessment" and its accompanying Assessment 
Handbook) LAMP will articulate a comprehensive set of guiding principles addressing 
issues related specifically to statewide assessment practices and policy. These principles 
will be used to make recommendations about the role of MDHE in assessment across the 
state, definitions of different forms and purposes of assessment and their relationship to 
state policy. 

2. LAMP's charge highlights the capacity of assessment to foster student learning, 
establish foundation for a culture of continuous improvement, and provide ways to 
demonstrate accountability. In the decentralized post-secondary educational environment, 
assessment provides an efficient and effective means of establishing academic trust 
among institutions, facilitating student transfer and ensuring educational quality. This 
process recognizes multiple opportunities for assessment to provide useful information 
along the path of an educational career. Reflective of the guiding principles, assessment 
at each transition point must ask questions related to purpose, audience, methods, and the 
role of the state and statewide collaboration, etc. LAMP is charged to create policy 
recommendations related to each point of transition. a. Access and Placement 
(Preparation) SB 389 recognizes a need for post-secondary institutions to establish 
expectations of student academic competencies (skills and knowledge) in order to 
succeed at collegiate level coursework. Through the Curriculum Alignment Initiative 
entry-level competencies for many areas have created criteria for access to college. 
Assessment related questions concerning how to determine attainment of these 
competencies need to be addressed. 

Potential Questions/Issues 
o What essential entry competencies important to access and college readiness have 
yet to be addressed by CAI? 
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o Howcan we best assess the entry-level competencies for entering postsecondary 
students? 

o What still needs to be done to align CAI Entry Level Competencies with DESE 
educational assessment standards like Course Level Expectations (CLE)? 

o Incases where CLEs are adequately aligned with entry-level competencies, are 
the End-of-Course examinations (EOC) of the CLEs sufficient to assess for access 
to postsecondary coursework? 

o What kinds of supplemental assessment are required if EOC's not sufficient 
and/or for exceptions like late transfer students, out-of-state students, 
advancement from remediation/developmental coursework to college level etc.? 

o Are competencies required across the board for all subject areas for access to any 
collegiate-level coursework or is performance considered on a subject by subject 
basis? 

o How do we ensure that Dual Credit students meet the same expectations as other 
students? 


b. Beginning General Education Course Transfer 

SB 389 addressed concerns regarding the transfer of single beginning general education 
courses for collegiate credit for those students not transferring with the 42-hour block of 
articulated credit or an associate's degree. Assessment related questions regarding the 
appropriate certification of credit given the development of course-based exit-level 
competencies in CAI need to be addressed 

Potential Questions/Issues 


o What are advantages/disadvantages of statewide exam in beginning general 
education courses? 

o What grading policies and procedures would have to be in place for grades to 
demonstrate achievement of exit competencies? 

o Howcan we respect institutional autonomy while ensuring the transfer of 
knowledge and skills, not just the transfer of credit? 

o Are there ways to "tune" learning goals or curriculum across the state so that 
grades might be sufficient demonstration of exit competencies mastery? 


c. College level General Education 

Assessment of general education competencies attainment provides a significant 
opportunity for intervention to promote student success, accountability to public 
stakeholders, and trust among institutions to facilitate transfer and articulation. LAMP is 
charged to develop a strategy that 1) enriches institutional practices which provide useful 
feedback for student and course improvement, 2) assures correspondence of student 
learning achievement across institutions, and 3)provides meaningful demonstrations of 
associated student learning for the public. 

Potential Questions/Issues 
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o What do we mean by general education (e.g., first two years of college, 
foundational content knowledge and cognitive skills, liberal education) what do 
we want to test for? 

o What are effective means of assessing general education for improvement of 
student learning that may also serve purposes of accountability reporting and 
institutional benchmarking? 

o What assessment policies and practices are necessary to facilitate transfer of 
credit (1) in courses where specific exit competencies have been specified, (2) in 
courses where specific exit competencies have not been specified, and (3) in the 
case of the 42-hour block? 


d. Major Fields 

Assessment in major fields ensures that institutions in Missouri are maintaining 
alignment with their fields of specialization, adequately preparing students to enter their 
chosen profession, and providing good stewardship of state resources. 

Potential Questions/Issues 


What kinds of reporting will provide sufficiently useful information for public policy as 
indicated in Imperatives for Change? What kinds of reporting and collaborative 
assessment and course configuration might extend beyond the IFC requirements. e. 
Licensure and Certification 

Results of licensure and certification also serves to prove good stewardship of state 
resources and indication that students are prepared to enter fields with criterion- 
referenced licensure. 

Potential Questions/Issues 

o Are licensures and certificates comparable across fields? What is viable 
reporting? 

o How do we gain more data from outside licensure programs? 

o What kinds of reporting will provide sufficiently useful information for public 
policy as indicated in Imperatives for Change? What kinds of reporting and 
collaborative assessment and course configuration might extend beyond the IFC 
requirements. 


f. Graduate level Access, Admission and Completion 

Missouri higher education has a significant interest in producing undergraduates ready 
for graduate study. Many assessment issues related to undergraduate access and 
completion may be related to graduate transition point as well. 

Potential Questions/Issues 


o Other than GRE Scores what kinds of assessments and reporting might inform the 
preparedness of undergraduates for graduate study? 


g. Workforce Competency 
Imperatives for Change asks for assessments of collegiate graduate performance in the 
workplace. Workplace assessments provide useful information to benchmark program 
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content and student achievement with the knowledge and skills required by employers. 
Potential Questions/Issues 


o What kinds of reporting will provide sufficiently useful information for public 
policy as indicated in Imperatives for Change? What kinds of reporting and 
collaborative assessment and course configuration might extend beyond the IFC 
requirements. 

o What kinds of useful feedback for institutions and programs, recent graduates, 
and state reporting would be helpful? 

Encourage and facilitate qualitative advancement of institution specific assessment 
practices through collaborative conferences, seminars, pilot projects, benchmark 
data collection and dissemination etc. 

Provide for a strategy for further review of assessment policy and evaluation of 
assessment practices across the state. 
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Assessment Culture and Practices 


Across Missouri Postsecondary Institutions 


(A draft report by the LAMP Assessment Practices Subcommittee) 


Executive Summary 

The Assessment Practices Subcommittee reviewed secondary and postsecondary assessments 
currently in use in Missouri. This draft report, Assessment Culture and Practices across Missouri 
Postsecondary Institutions reflects upon the infrastructure and general role of assessment 
practices at Missouri institutions. Below is a summary of the major points outlined in the 
subcommittee’s report. 


What do we know about current Missouri practice in assessment? 





e Stakeholders who perceive sustainability as unlikely are reluctant to invest in one policy or 
process. 

e Institutions would like MDHE to provide more data, research, and coordination on important 
assessment issues, especially placement. 

e Institutional assessment officers believe collaboration around best practices and statewide 
policies can increase the reliability, validity, and application of assessment. 

e All institutions are engaged in some form of student assessment, though there is great 
variation in the instruments used, administrational infrastructure, and the extensiveness to 
which individual students are assessed. 


O 


O 
O 


Institutions assessing students’ basic skills in general education use one of five 
instruments: CAAP, CLA, MAPP, C-BASE, or Work Keys. 

Assessment within the academic major is primarily done using ETS Major Field tests 
and/or institutionally designed cap stone courses and comprehensive course 
examinations. 

Concerning placement most community colleges use COMPASS. 

Assessment of affective development, attitudinal surveys, institutional effectiveness, 
and first year student experience is sporadic. 


e Institutions have made expansion of assessment programs on their campus a priority through 
multiple avenues and by engaging critical stakeholders: 


O 


Institutions have expanded their assessment programs to fulfill mission objectives and 
respond to public calls for increased transparency. 

Assessment has become a central aspect of institutional mission and practice. 

As a criterion of success, the role of faculty has become central not only to the 
implementation, but also the planning, structure, and decision making process of 
assessment on campuses. 

Almost all institutions believe that faculty are invested in assessment but this 
relationship needs continual nurturance and development. 
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o Technological advances (dashboards, assessment software like Foliotek and Weave) 
have increased the participation in and effectiveness of assessment on campuses, but 
more investment is needed in this area. 


Preface 

This document is a preliminary report outlining the state of assessment culture and practices of 
post-secondary institutions across Missouri. The report was commissioned by the Learning 
Assessment in Missouri Postsecondary Education Advisory Council (LAMP) and produced by 
the Assessment Practices Subcommittee (APS). LAMP’s charge focuses upon analysis of 
current and future needs in Missouri to increase the quality of education through learning 
assessment. This report was commissioned to provide context for LAMP’s policy 
recommendations to the Commissioner of Higher Education. 


Assessment of student learning is an evolving discipline. While institutions draw from a 
common battery of instruments to measure student learning, student satisfaction, instructional 
quality and institutional effectiveness; the methodology, analytical perspective and application of 
the information varies significantly across institutions. Further, many important aspects of 
assessment are not related to the instruments but to infrastructure, stakeholder attitudes, and 
relation to state, federal, and accreditation requirements. 


Missouri has a long and productive history encouraging assessment and collaboration across 
institutions. Along with the critical contributions of the Missouri Assessment Consortium 
(MAC) found in 1991, strategic plans and reporting have consistently contained explicit calls for 
improvement in student learning and relevant assessment measures. Improvement of student 
learning and effective assessment for instructional improvement and accountabilities are 
paramount priorities for the Department of Higher Education. The strategic plan, Imperatives for 
Change reinforces these priorities by reporting performance on general education, major fields, 
and licensure and certification examinations. 


The depth and breadth of assessment practices should not be underestimated. At the time of 
LAMP’s inception in October 2008 scarce information about these assessment practices was 
available. This report begins to address this deficit. The report will draw upon two primary 
sources, 1) the Missouri Assessment Instruments Survey (MAIS) which included questions 
concerning the availability of an assessment plan and solicited information about the use of over 
60 different types of assessment instruments, and 2) the Survey of Assessment Culture (SAC) 
concemed with information related to the infrastructure, attitudes, and general assessment 
practices. Additionally, respondents were invited to share opinions about the effectiveness of 
current state policies related to assessment, and how they might be improved’. 


Assessment Planning and Institutional Mission 

Almost all institutions have a formal plan that is publicly accessible while only about half of 
these plans are available on the institution’s website. The SAC further inquired about what kinds 
of assessment are explicitly referenced in the assessment plan. Of the 21 institutions with an 





* For more information on the methodology of the SAC and MAIS data see Appendix A. 
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available plan 100% referenced institutional level assessment, 90% program level, 81% course 
level, and 71% entrance or preparation level assessment. 















Formal Plan On Website | Publicly Accessible 
Plan 
8 89% 3) ove |S 33% 
fl 79% 7 50%] 10 91% 
10 91% Sy 23% 9 82% 


85% 53% 69% 





MAIS SAC 


Formal plans play an important role in setting clear objectives and processes. For many 
institutions assessment is a vital component in ensuring that instructional and administrative 
decisions throughout the campus are aligned with stated mission ad values. Institutions: 


e Have clearly connected assessment to their institutional mission, and it plays a clear part, 
at least on the institutional level, in evaluating performance. 

e Have performed formalized linkages between the institutional mission and the assessment 
of program, department, course, or placement level student learning outcomes. 


Infrastructure 

Infrastructures that support assessment responsibilities vary greatly across institutions. 
Capabilities and limitations are related to more than just the value associated with assessment. 
Institutional size and budget, mission focus, and non-assessment organizational structure 
significantly shape the support systems of assessment. Respondents were asked to —dscribe the 
infrastructure and resource allocations dedicated to student learning and assessment (i.e. 
academic support centers, research offices, committees, strategic plan, faculty involvement, 
professional development, software applications etc.)”. 


e Management of assessment responsibilities varies. Ultimate responsibility is most often 
at a senior academic level or office of assessment or institutional research director; but 
faculty also play important roles as assessment directors. 


Public 4 Public 2 Independent Total 
Year Year 4 Year 


3 Z 
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*“ Assistant” includes the title Associate as well 
*CAO includes the titles Vice President of Academic Affairs and Provost 
*Office Director includes directors of Assessment, Institutional Research, and Learning Centers 





e New software applications have been developed and implemented to improve tracking of 
the large amounts of data associated with student learning objectives. These applications are 
credited with significant improvement in data management, faculty and student participation, 
and instructional improvement. 

e Innovative assessment activities and professional development opportunities increase 
faculty proficiency in assessment skills. 

e Development of a detailed assessment strategy and infrastructure for many institutions 
has generally resulted in increased support and participation in assessment by all collegiate 
stakeholders. 


Faculty Support 

Achieving a quality assessment program requires the integration of supportive faculty in the 
process. Yet assessment may be a challenge for faculty concerned the additional responsibilities 
take away from teaching, add to the overall workload, and are not proven solutions. In spite of 
these often voiced concerns, all institutions reported broad faculty support and participation by 
faculty. The most negative comments suggested faculty support was —miad”, -growing but not 
everyone was on board”, and -some faculty resistance, but not strong”. Most assessment 
programs are beginning to rely upon faculty more and more through course embedded 
assessments and program reviews. Faculty are generally cited as integrated into the assessment 
program. 


Student Support 

Student support and participation also receives high remarks from survey respondents. Several 
cited high participation at assessment events and response rates for -eptional” surveys like the 
National Survey of Student Engagement (NSSE). While assessment has become a norm in much 
of student life, some have noticed that enthusiasm begins to flag as students become seniors. 
Others have heard students complain about passing a general education test in order to graduate. 


Assessment Instruments 

One primary function of the Missouri Assessment Instruments Survey was to inventory the 
instruments used for student learning improvement and other levels of assessment. The APS 
created a follow up survey to this study which moves beyond a simple inventory requesting 
information on methodological implementation, collection and analysis strategies, and how the 
data is used. This survey is currently being field tested and may be employed later in 2009 to 
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contribute more information to this process. Below are some of the key findings from the MAIS 
responses’: 


Instruments primarily used to measure the general cognitive ability of students and assess 
apprehension of general education learning outcomes were classified as —Basic Skills”. 


e Every Public Institution and all but one Independent institution indicated the use of a 
Basic Skills Assessment. Many also report these scores through the Voluntary System of 
Accountability (VSA). 








CAAP CLA MAPP CBASE GE _ Workkeys 
0 2 2 5 
9 0 3 11 9 
3 3 d 8 
12 5 12 24 9 


e Due to an earlier pilot project with the CLA in 2002-4 numerous institutions indicated 
past usage but now used the CAAP or MAPP. In fact, of reporting institutions, only 5 of 
the 15 institutions that had used CLA in the past 5 years intended to use this year or in the 
future. 

Major Field exams are comprehensive instruments measuring undergraduate understanding of an 
entire field of study. 
e All Public four year and many of the independent institutions use Educational Testing 
Service’s (ETS) Major Field Tests. 
e Many institutions also use institutionally designed capstone courses (65%) and 
comprehensive course examinations (50%) for critical fields assessment. 
Placement examinations consist of instruments designed to evaluate an incoming student’s basic 
skills for the primary purpose of collegiate course placement and advising. 
e COMPASS is the preferred placement tools among the public community colleges 
(86%). 
e In spite of growing remedial needs among baccalaureate conferring institutions, few 
indicated the use of Accuplacer, ASSET or COMPASS. 


Many institutions supplement external evaluations of student cognitive capacity and growth with 
instruments used to measure adaptation to the higher education environment, religious identity, 
demographic, attitudinal and other subjective and affective dimensions. Many such instruments 
overlap in purpose with the following category of institutional effectiveness. While those 
primarily tasked to evaluate the affective development of students are used less frequently, 
several of these instruments were indicated by surveyed Missouri schools. 


e Only 11 (32%) of respondent institutions indicated current use of an instrument assessing 
some dimension of affective development. 





. Appendix D contains a copy of the final summary tables from the MAIS report. 
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e The Cooperative Institutional Research Program’s Freshman and Your First College Year 
(YFCY) surveys were the most commonly used instruments. While only 7 institutions 
claimed current usage, another 8 had used one of them in the past and 4 are considering 
for future use. 


Numerous instruments exist to evaluate student, faculty, staff, alumni, and other interest groups 
opinions or institutional experiences. Because of the quantity and variety of instruments 
available there is a larger pattern of disjointed use between the past, present, and future than 
among other categories. 
e Most four year institutions have used the National Survey of Student Engagement at 
some time although only 29% currently participate and these are mostly Independent 


colleges. Among institutions not currently participating almost all are considering future 
use. 


e After NSSE, Noel Levitz’ surveys like the Student Satisfaction Surveys are the most 
commonly cited instrument. 


e 50% of all institutions and 73% of public 4-year universities administer a self-designed 
senior exit survey. 


e Institution designed measures of institutional effectiveness are also commonly employed 
among alumni, faculty, and staff. 


Five final categories related specifically to First year and Prospective students: Health Fields 
Assessment, Data Warehouse participation, course evaluations, and miscellaneous are included 
in the appendix tables. While some of these tests overlap in purpose with earlier categories their 
unique populations (first year, health professionals etc) or scope of administration, warrant 
separation. 

e 4 independent and 7 public institutions indicated using some kind of first year or 
prospective student survey apart from those targeting affective development. 

e 44% of institutions passed information to the National Student Clearing House, while 5 
of the 8 independent four year schools participated in the University and College 
Accountability Network (UCAN) and 6 of the 13 community college systems (43%) 
participated in the National Community Colleges Benchmarking Project (NCCBP). 

e 8 (73%) of public four-year institutions use Portfolios or ePortfolios in Assessment work. 
These were not included in the basic skills category because their purpose an 
implementation varies greatly. 


Perspectives on Assessment Needs 

Assessment continues to be a developing discipline. Survey responses illustrate that assessment 
professionals at Missouri’s institutions are committed to enhancing the methods and structure of 
assessment to advance the goal of improving student learning outcomes. This professional 
dedication is reinforced by the requirement of governing boards and accreditation bodies to 
consistently review assessment plans and implementation. It is not surprising then, that only one 
institution did not offer any ideas when asked —What is the single most important change or 
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improvement your institution could make to increase the quality and effectiveness of student 
learning assessment at your institution. Several central concerns dominated these responses. 


Institutions assessment and research offices need more personnel and resources. 

Many institutions lack the coordination of a full-time assessment coordinator would 
provide. Such a coordinator could implement programs on assessment and work with 
faculty on teaching, learning, and assessment. Along with this concept, smaller 
institutions would like more data and strategic support with the addition of a dedicated 
research office and support center for assessment training and resources. Beyond the 
infrastructure support personnel, other institutions admitted a need for a more structured 
and organized system of assessment overall. 

Implementation of assessment policy was also identified as an area for improvement. 
When explicitly asked about potential changes, some institutions suggested that 
assessment needs to become a normative component of the educational process for 
students and faculty. If students were required to complete general education 
requirements within a certain time frame, pre- and post-testing would allow for better 
measurement, intervention and assist in both student feedback and instructional 
development. Many course and program assessment practices are the responsibility of 
faculty without significant oversight, incentives or consequences for —non-completion. A 
more defined process requiring course embedded assessments, structured review and 
feedback, and program review would enhance the success of students at learning 
outcomes. The next step then is to close the loops in assessment programs by ensuring 
programs use the collected data to analyze and make responsible changes to the 
programs. 


Structured assessment policies require not only significant participation by faculty, but 
embedding the practices in the experiences and knowledge of faculty. The reference group felt 
that overall support of assessment would derive from the group spending the time, effort, and 


resources to gain the support and understanding of reluctant faculty. 


Best Practices in Missouri 

Although improvement is always possible, Missouri institutions excel in many areas. Many 
institutional representatives highlight the capacity and need for MDHE to be a coordinator of 
data, information and best practices. Responding to this sentiment, the APS committee invited 
institutions to share some of their institution’s best assessment practices. Three dominant issues 
underlie many of the solutions. 


The need for improved alignment between assessment practices and institutional mission 
and objectives 


How to increase faculty, staff, and student involvement and encourage an environment 
where assessment is normative 
Developing assessment practices with increased reliability, validity, and meaning. 
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Mission Alignment 

Several institutions indicated they have initiatives to align student learning outcomes objectives 
with the institutional mission. More than one respondent claimed these initiatives as part of their 
AQIP? action project. Other institutions highlighted the benefit of requiring annual assessment 
plans for each department be reviewed by an assessment committee and relevant performance 
tracked. Implementing a five year review of programs also allows for a tighter alignment of 
mission and values with instruction and learning. 


Assessment Participation 

While well articulated plans, and alignment of objectives and missions are critical for continual 
improvement in student learning implementation is impossible without —by-in” from staff, 
faculty and students. Several of the best practices cited by institutions demonstrate successful 
strategies to resolve this problem. The creation of assessment areas and dashboards on campus 
intranet has received a clear testimony of success. These kinds of enhanced communication 
strategies increase awareness, accessibility and utility of information, and even encourage timely 
completion of assessment plans. Other assessment tracking applications like Foliotek, Weave, 
and other Eportfolio programs also encourage participation from students and faculty and allow 
linkages with student information systems to enhance intervention capabilities. 


Technological solutions are one strategy to increase participation and create a normative 
assessment culture. Other respondents underscored the benefits of structural and testing 
strategies. A smaller liberal arts college transferred everyday assessment responsibilities from a 
college committee to departmental assessment coordinators. For an institution without many 
administrative staff this created greater involvement of departments in the process. Another 
campus requires faculty to give at least one assignment to each class that aligns with established 
rubrics. These are then reviewed by an external committee to evaluate student learning and 
quality improvement. 


Incentives and collaborative sharing can also be an effective strategy to increase participation in 
assessment testing and professional development. Several institutions have created Assessment 
Days where students and faculty are given class release to complete assessment instruments. 

One community college reports 75-90% student participation. Free food for lunch and door 
prizes appear irresistible for students. Similar assessment days for professional development and 
assessment showcases where departments can share best practices within the campus community 
may also increase faculty involvement and competence. 


Assessment Quality 


> The Academic Quality Improvement Program (AQIP) of the Higher Learning Commission is an alternative five-year 
re-accreditation process prioritizing continuous quality improvement processes with institutionally designed 
objectives and self-assessment. 
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In addition to clear articulation of assessment goals and participation from students and faculty, 
institutions must also develop processes that provide quality information to enhance student 
learning and program quality. Some related best practices include detailed analysis and feedback 
based on a global application of Major Field Tests, comprehensive testing for academic and 
writings skills, measuring student outcome performance at the entry, midway, and exit of 
collegiate life, general education capstone experiences and general education portfolios. 


Institutional Perspectives on Assessment Related Policies 

Missouri’s on-going efforts to improve student learning and educational opportunity across the 
state have generated several polices related placement, remediation, curriculum alignment, 
transfer, and educational proficiency. These policies are fundamentally related to student 
learning improvement and will likely be affected by changes in assessment policy and practices. 
Institutional representatives were asked to provide feedback on the current status of these 
policies and perspectives on what was required for success. 


Placement 

Twelve institutions indicated they would like the state to move towards further standardization of 
a placement policy. Among these institutions eight explicitly requested an established range of 
placement scores. Eight of the remaining twenty-one institutions asserted that policy should 
prioritize individual institutional policies. Another common call was for MDHE to collect and 
disseminate placement data and best practices. Other significant suggestions included a desire 
for a standardized K-12 College Preparation instrument; a requirement that all sophomores and 
juniors take the ACT; mandating institutions to subscribe and implement clear entry level course 
expectations; and ensure the quality of dual credit programs. 


Remedial and Developmental Education 

Comments related to remedial and developmental education were limited. Only a few 
respondents indicated MDHE policy should move towards standardization or collaboration. 
These were balanced by a few explicitly expressing state policy should prioritize institutional 
policy. Institutions identified the need for MDHE to continue collaboration with the K-12 sector, 
increase the visibility of remediation and developmental needs, design a common placement 
instrument, collect and disseminate data and information, and ensure appropriate funding. 


Curriculum Alignment Initiative (CAI) 

Many institutions feel that the current CAI approach is adequate and should be maintained. 
There were several specific suggestions for a common course numbering system, as well as, 
increased alignments and cross validation between CAI competencies and other sectors like 
DESE, LAMP findings, and across institutions. 
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Transfer and 42 Hour General Education Block 

The legacy of the 42 hour general education transfer policy continues to be an important issue for 
many institutions. Many of the community colleges would like to see the policy mandated so 
certified students need not worry about its transferability. Others suggest some revisions like 
increasing the block to 60 hours, creating a common assessment tool, and the development and 
alignment of general education competencies. 


Assessment of Major Fields 

Of all the policy areas, the strongest push for prioritizing institutional policies was related to 
Major Fields. While a few advocated for more standardization, and slightly more that MDHE 
encourage collaboration and dissemination of information, most cited the diversity and 
uniqueness of individual programs, and that assessment of Major Fields is already governed by 
accreditation and licensure. Other comments included the desire that state licensure and 
certification should be aligned and inherited from regional accreditation bodies, and for further 
alignment of DESE and DHE policies related to technical skills assessment. One institution 
suggested MDHE create incentives for rewarding institutions and students for success, and 
another for the integration of Workkeys into the state policy for community college assessment 
of major fields. 


Institutional Suggestions for Consideration 

An important early benefit of LAMP has been the sharing of problems, solutions, concerns, and 
ideas for the future as participants pay careful attention to common assessment issues and how 
public policy might assist institutions. An important role for MDHE is to facilitate collaboration 
among diverse institutions. The complexity of assessment practices and use necessitates such 
partnerships both to learn from others as well as address cross-institutional issues like transfer, 
college preparation, and current statewide policies. Responding to: -n which types of 
collaborative projects related to student learning assessment would your institution be interested 
in participating?” Institutions highlighted the collaborative advantage to: 

e Sharing best practices and learn from the innovation and success of other institutions. 

e Inform and influence policy issues that affect all institutions like dual credit, specific 
tasks and methods like measuring and advancing critical thinking, problem based 
assessments, reducing remedial coursework, partnering with secondary schools to 
reinforce preparation and transition, and creating CLA content specific practice 
examples. 

e Sharing data for cross-institutional research into topics such as: the success of students at 
different placement score cutoffs, tracking of student transfers (e.g. the recent UMSL- 
STLCC transfer project), and following wage and employment data of graduates. When 
asked about how MDHE could improve institutional assessment capacity and quality data 
collection and support were central. 

e Increase assessment quality with partners to cross-validate assessment tools and student 
learning objectives, especially around CAI. Partnering to create common discipline 
based goals and objectives or redesign entire curricular areas for student learning 
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improvement provides advantages for transferability and education commensurability. 
This approach allows for faculty autonomy in course design while promoting transfer and 
articulation. 


In addition to providing data resources and facilitating collaboration, assessment officers asked 
for additional financial resources committed to assessment. Funding was requested for testing, 
resources, and to support the introduction of innovative methods, materials, training, and 
equipment to engage students more fully in learning activities. 


Conclusion 

Institutions in Missouri are committed to improving student learning. The growth of assessment 
practices and their integration into the administrative and pedagogical fabric of academic life 
reflect the importance of assessment to provide information for faculty guidance, curriculum 
modification, administrative performance review, institutional mission alignment, public 
accountability and to provide students with information about their academic growth. 
Institutions have expanded their assessment programs to fulfill mission objectives and in 
response to periodic waves of public interest with corresponding legislative mandates and civic 
initiatives. In recent years, the wealth of institutional and state-wide practices and policies has 
been reinforced by a quality improvement focus by the accreditation process of the Higher 
Learning Commission (HLC). The practice of assessing student learning in Missouri is 
improving as well. 


Four dominant themes run throughout the responses to the survey. First, assessment has become 
a central aspect of the educational life, strategic plan, and mission of many postsecondary 
institutions. Second, there is an incredible diversity in the structure, hierarchy, and practice of 
assessment across Missouri postsecondary institutions. Third, in spite of this diversity there is a 
strong commitment by institutional assessment officers to build consensus and collaborate on 
assessment policies that may enhance student learning across the state and quality in each 
institution. Finally, continued investment in infrastructure and program development at the 
institution and state level is needed to achieve student learning improvement. 


Institutions across Missouri are integrating continuous improvement into their institutional 
mission and objectives. Almost all institutions have deliberate institutional and program level 
assessment plans and nearly three-fourths have formal course and entrance or preparation 
policies. This commitment is beginning to be reflected in strategic, facility, and infrastructure 
planning. While the process is not complete at any one institution, the values of improving 
student learning are becoming a recognized language of educational administration and 


pedagogy. 


While the principle of quality improvement becomes more pervasive, the implementation and 
even underlying philosophy varies from institution to institution. In most cases this variance 
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results from responses to institutional-specific values and conditions. This range of practice 
reflects the diversity in institutional culture across the state, as well as, the nascent development 
of assessment practice and the relative isolation in which solutions are created. 


As assessment practice becomes more established, assessment professionals across the state 
highlight the opportunity for cooperation to further institutional and state-wide improvement. 
Respondents indicated a need for further sharing of best practices and discussion of common 
challenges. Many institutions recognize a need to create refined common policies to promote 
trust and address larger social educational problems like educational mobility, remediation and 
developmental education, and workforce preparation. Institutions also have reservations about 
any policies which might over-ride their flexibility to address these same issues within their local 
context and mission. 


With a few exceptions, institutions would like MDHE to provide more data coordination and 
research related to important assessment issues, and more coordination of collaboration on these 
issues and with other organizations like DESE and state licensure boards. There is a particular 
recognition that collaboration can increase the reliability, validity, and meaning of current 
assessment practices. 


In addition to the integration of assessment values into the mission and strategic plan, many 
institutions indicate that the success of assessment programs is tied to the prioritization of 
infrastructure needs by the administration, and the investment of faculty in the process. 
Technological advances (dashboards, assessment software, data warehouses, etc) have increased 
participation and the effectiveness of assessment on campuses, yet more investment is needed in 
this area. Faculty have become central not only to the implementation, but also the planning, 
structure and decision making process of assessment on many campuses. Because faculty are 
central to success, institutions have focused upon placing them at the center of the process. This 
investment needs to be continually nurtured and reinforced. 


Survey responses show that some disagreement over the scope and nature of statewide 
assessment policy results from philosophical differences in the role of assessment or the 
relationship of the state and individual institutions. As the Assessment Practices Committee has 
discussed the survey results among themselves and with other faculty, institutional 
administration and assessment professionals, two further reservations also predominate. 
Faculty, as well as, institutional administration and assessment professionals, is wary of an 
increased workload associated with further assessment programming. Secondly, stakeholders 
may be reluctant to invest in one policy or process only to find their efforts quickly superseded 
by subsequent initiatives. Participants in this discussion insist that policy development must be 
mindful of these contexts. 


22|Page 


The high survey response rate and investment in the LAMP process are testimonies to the spirit 
of cooperation thriving among assessment professionals across the state and their desire to 
enhance the discipline and benefit Missouri higher education. This report provides substantive 
evidence about current assessment practice and culture to inform future policy discussion. More 
importantly it demonstrates that both the need and will are present to improve student learning 
outcomes and the quality of higher education in Missouri. 


Appendices 


Document appendices can be downloaded from: 


www.dhe.mo.gov/files/lampassessmentculturesurveyappendices.docx 


Appendix A: SAC and MAIS Methodology 
Appendix B: SAC De-Identified Responses 
Appendix C: SAC Instrument 

Appendix D: MAIS Survey Summary Tables 
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Communications Subcommittee 

Activities 

The Communications Subcommittee developed and implemented a plan to foster communicating 
internally and externally by developing a LAMP Primer detailing the genesis of the LAMP 


Advisory Council, as well as creating two newsletters, in December 2008 and May 2009, 
updating participants and interested constituents on LAMP’s activities. 
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Missouri Department of Higher Education 





May 2009 


Contents What is LAMP? 


What is LAMP? os ; 
For some of you reading this message, you may be asking yourself: 


What does LAMP stand for? What exactly is LAMP? What is this group doing? What will be the 


i i ? 
LAMP’s Genesis impact of this work? 


Recognizing that there are different levels of understanding about the 
work of LAMP, we have developed a primer that outlines the genesis 


Why you should be interested in and intent of the LAMP initiative. 
LAMP 


LAMP Goals 





LAMP: Myths and Facts 


What does LAMP stand for? 


LAMP stands for, “Learning Assessment in Missouri 
Postsecondary Education”. The Co-Chairs and organizers of the 
group considered several different names for the group, but 
specifically chose the acronym LAMP, because of its significance 
to the group in its role to provide information and data to policy 
makers to illuminate their decision-making processes and assist 
them in making informed policy decisions. 


LAMP Webpage: 





Upcoming Meetings 


LAMP’s Genesis 


LAMP was initially conceptualized and initiated by the Missouri Department of Higher 
Education (MDHE) to dialogue and provide research, data, and recommendations to the 
Commissioner regarding assessment-related policy issues. Three institutional representatives 
from across the educational sectors were chosen to provide grassroots leadership, with 
support from MDHE staff. The intent of the group is to bring the experience and expertise of 
institutional representatives and practitioners to the table as the MDHE begins to consider 
appropriate ways that assessment be considered from a state-level perspective. 


LAMP Drivers 


Two initiatives underway at the state level have driven the creation of LAMP to 

consider assessment in the context of higher education policy: the Curriculum Alignment 
Initiative (CAI) and the CBHE statewide coordinated plan for higher education—Imperatives 
for Change. 


The Curriculum Alignment Initiative, initiated in 2007, has produced competencies that 
outline knowledge and skills necessary for success in collegiate-level coursework and for 
completion of beginning general education courses. While CAI has undertaken the 

work to develop the competencies, it was the charge of the group was also to identify the 
policy impacts for a larger group of both content and assessment professionals, LAMP, to 
consider. To learn more about the history and current status of CAI, visit: 
http://www.dhe.mo.gov/casinitiative.html. 
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LAMP Primer, Page 2 


The Missouri's statewide coordinating plan for Higher Education, Imperatives for Change (IFC), was adopted by the Coordinating 
Board for Higher Education in July 2008. IFC provides a vision that has been developed collaboratively by Missouri's higher 
education institutions and the Coordinating Board for Higher Education. This plan will serve for the next three to five years as a 
foundation for prioritizing goals, justifying an increased resource base and allocating resources. This plan has, as one of its main 
goals, the need to assess student learning outcomes through multiple venues. LAMP has been charged with considering 
assessment surrounding these areas. To learn more about the IFC, visit: http://www.dhe.mo.gov/ifc.shtml. 


LAMP Goals 


The central goal of LAMP is to “thoroughly and thoughtfully explore assessment-related policy questions, based upon current 
practice and literature surrounding these issues, and provide the resulting considerations and recommendations to the 
Commissioner for use in the development of policy”. 


The work of LAMP has been shaped by three documents: the LAMP Charge, the LAMP Policy Guidance, and the LAMP inclusion 

values: 

e The LAMP Charge was put forth by the Commissioner outlining the tasks and end report that LAMP was asked to deliver. This 
document framed the reason for forming the group, the importance of gathering professionals from across domains, and the 
specific areas for inclusion in the final report. 

e The Policy Guidance document clarified the policy questions, driven by CAI and IFC, that the MDHE needs addressed in order 
to develop assessment related policy. These were meant to frame thinking and direct LAMP to specific policy areas. 

e The LAMP Inclusion Values were developed by the LAMP members themselves, as the method and process for moving 
forward with the work of LAMP was under the purview of the group itself. This document seeks to articulate the process 
values the group would uphold in moving forward with the work. 


These three documents have served to clarify and direct the actions of the group. 


Why you should be interested in LAMP 


LAMP is an opportunity to bring together individuals across the state with experience and expertise in multiple domains, and 
engage in dialogue and discussion with the express intent of bringing quality information to senior officials and upper-level 
administrators tasked with developing assessment-related policy. The LAMP process is an open opportunity to collaboratively 
identify the issues, challenges, and prospects for policy development. There is no expectation that participants must be in 
agreement, but that as a group LAMP can illuminate the critical issues for consideration in developing effective assessment policy 
for the state of Missouri. 


LAMP: Myths and Facts 


MYTH: LAMP is making policy regarding assessment. 


FACT: LAMP does not make policy. The group is charged with exploring assessment practices in Missouri and professional 
assessment literature in order to provide information and recommendations to the Commissioner. This information will be utilized 
by MDHE to make informed, data-driven decision making and policy development. The Commissioner and MDHE Senior Staff 
develop policy in consultation with institutional leadership; these policy recommendations are then brought forth to the 
Coordinating Board for Higher Education for consideration and action. 


MYTH: LAMP will constrain or devalue institutional autonomy. 


FACT: LAMP’s purpose is to critically look at the policy issues faced at the state level surrounding assessment and provide quality 
information to policy makers about the issues and concerns that need consideration in the development of thoughtful and 
effective policy. Members of the group are not being asked to all agree on the same thing; indeed, there are many opinions about 
possible policy solutions. It is the synergy of bringing together so many individuals that will allow LAMP to provide thorough and 
nuanced recommendations to these assessment policy issues. 


MYTH: The purpose of LAMP is to produce statewide assessment tests. 


FACT: LAMP is charged with examining best practices and professional literature, and dialoguing about that information as a group 
in order to identify areas of critical consideration and recommendations for policy consideration. There is not any foregone 
conclusion regarding assessment put forth to the group; MDHE is seeking information to better inform the development of policy 
and the appropriate use of assessment. 
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MYTH: The MODHE will not take LAMP recommendations into consideration. 


FACT: The Commissioner and MDHE are committed to promoting collaborative dialogue and sharing policy development. LAMP 
was expressly initiated to utilize the expertise available in the state in the area of assessment, so that policy development and 
policy decisions might be better informed. It is of course, no guarantee that all recommendations will be acted upon, but they are 
of great value in bringing forth the critical areas of concern in the policy development process. Good policy cannot be created ina 
vacuum. 


MYTH: These issues are too big for LAMP to consider. 


FACT: Indeed, these are significant and complex issues for the group to consider, especially considering the breadth of the field of 
assessment. However, the MDHE has provided guidance as to the specific domains within assessment with which they are 
requesting assistance and information, as outlined in the Policy Guidance document. In addition, Commissioner Stein recently 
wrote a letter requesting the LAMP group specifically focus on completing recommendations regarding access and placement for 
the June 2009 report, with the understanding that LAMP would continue to work on other policy recommendations after that date 


MYTH: There is no way LAMP will be able to make recommendations by the June 2009 deadline. 


FACT: Early on, LAMP members, with support from the Commissioner, decided that given the number of policy areas that the 
MDHE was seeking input and recommendations on that it would be necessary to limit scope the first year. The group decided it 
would focus their attention initially on three areas of primary interest: access and placement; beginning general education course 
transfer; and college level general education. Work has progressed on all three fronts, but with the Commissioner’s request to 
prioritize access and placement, it has been decided that LAMP will provide a report on progress to date for all group activities but 
provide a comprehensive analysis of critical issues and factors only in the area of access and placement for the June 2009 report. 


MYTH: Policies resulting from LAMP’s recommendations will force all institutions to have the same admittance policies. 


FACT: As mentioned above, LAMP is not a policy making body, and therefore would not have the authority or power to enact such 
policies. In addition, since LAMP’s members come from the institutions themselves, their dialogue and recommendations are 
based out of their experience and expertise as well as examination of practice and literature; it does not seem likely that LAMP 
recommendations would include such draconian suggestions. 


Volunteer to Participate with LAMP 


If you are interested in participating in LAMP, please email Angelette Prichett at 
. Your experience and expertise are welcome! 








www.dhe.mo.gov « info@dhe.mo.gov 
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MZHE LAMP UPDATE 


Missouri Department of Higher Education 


December 2008 


Welcome! 

Welcome to our new Learning Assessment in Missouri Postsecondary Education (LAMP) 
newsletter. This newsletter will be sent to all LAMP members regularly to update you on 

; whole group and subcommittee progress. We hope this will keep all participants informed 
tee Upaates: and up to date on our activities. 















~ 


pe eS Thank you for your participation and efforts to inform assessment policy in Missouri. 


Next Steps Group: Bringing LAMP into Focus 


At the conclusion of our first meeting in October, a small group 
called the “Next Steps” group was formed with the purpose of 
taking the dialogue from the first meeting and clarifying the 
purpose and direction of LAMP for the entire LAMP 
committee. 


Through feedback from the MDHE and through several 
meetings, the Next Steps group was able to develop several 
documents that outline LAMP’s goals and together form a set 
of framing documents that will guide our work: 





1. MDHE Policy Guidelines: This document, developed by MDHE, clearly outlines the 
Upcoming Meetings policy issues and questions that are a priority for the development of state-level 

a % policy in assessment. These areas that the LAMP group must focus in order to 
provide input and expertise to MDHE in developing informed policy. 


rie st ed Rites 2. LAMP Charge: This document was made available at our first meeting and clearly 
iterature Review Subcommittee outlines the drivers for the formation of our group, the rationale for our structure, 
and the deliverables that the MDHE has asked for by June 1, 2009. If you have not 
had a chance to review this document in detail, please take this chance to review 
all} it now. 


December 12, time 3. LAMP Principles of Assessment Inclusion: This document, developed initially by 

the Next Steps group and refined through all three subcommittees, establishes the 

values that will guide us as we move forward with our work. When new issues or 

Assessment Practices questions arise, the group will consider these fundamental principles as the basis 
Meteeliiitiacs: of our response. 


December 10", 9-10 ar 4. LAMP Timeline: Also presented at our first meeting, this document outlines the 
critical time points in order to deliver the final report in June 2009. 


To view these policy documents visit: http://www.dhe.mo.gov/lamp.shtml. 


Decerines 15 Oar 2B LAMP Subcommittee Updates 


LAMP participants were asked to join one of the three 
working subcommittees that have been assigned all tasks 
associated with LAMP. For those few remaining participants 
who may not have chosen a group, it is important that you 
chose a group (or groups) in order to participate in LAMP 
activities. The large group will come back together later in the 
process to evaluate subcommittee reports and develop policy 
recommendations from that information. 
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All three subcommittees met during the week of November 17-21; all three groups spent a 
portion of the meeting discussing and refining the framing documents before addressing 
group-specific work. Summary of major subcommittee activities are below; full meeting 
summaries are available at http://www.dhe.mo.gov/lampminutes.shtml. 


Communications Subcommittee: 
The Communication Subcommittee met on November 18”. Discussion included: 


overall communications strategy, including targeted messages and timing 
identification of both internal and external constituent groups 

recruiting additional members from unrepresented sectors 

keeping LAMP participants engaged 

keeping constituent groups informed, including assignment of liaison roles 

drafting a communications action plan 

developing a newsletter to keep all LAMP participants informed of subcommittee 
progress 

The group will meet face-to-face on December 18" to finalize an action plan and progress 
on communication materials 


Assessment Practices Subcommittee: 


The Assessment Practices Subcommittee met on November 20". 


1. The group decided to move forward with an initial questionnaire to move beyond the 
MAIS survey to how institutions assess different levels and categories of their students, 
programs, units, etc., with a target to send out by mid-December. 

2. Questions will focus on the policy areas of priority as outlined in the MDHE Policy 
Guidance Document. 

3. The questions are currently under development and review 

4. The group will finalize their action plan at their next meeting. 


Literature Review Subcommittee: 
The Communication Subcommittee met on November 21%. Activities included: 


® Discussion of role and scope of the subcommittee 
e Formation of literature review groups based on the policy areas of priority as outlined in 
the MDHE Policy Guidance document: 
o Access and Placement 
o Beginning General Education Course Transfer 
o College Level General Education 
© The group will meet weekly via conference call to update on progress and develop and 
finalize an action plan by December 15". 





What’s Next? 


With work well underway in all three of the LAMP Subcommittees, each group will continue 
to work on their respective tasks and projects. LAMP Co-Chairs and MDHE staff will identify 
a date in early 2009 to come back together as a large group to update on progress and get 
feedback on subcommittee activities to date. 
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rSriE LAMP UPDATE 


Missouri Department of Higher Education 


May 2009 





Hello! 


Welcome to the second installment of the Learning Assessment in Missouri 
Postsecondary Education (LAMP) newsletter. This newsletter is sent to all LAMP 
members and other stakeholders on a periodic basis to provide updates on whole 
group and subcommittee progress. We hope this will keep all participants informed 
and up to date on our activities. 


Thank you for your continued participation and efforts to inform assessment policy 
in Missouri. 


What is LAMP? 


For some reading this newsletter, you may be asking 
yourself: What exactly is LAMP? What is this group doing? 
What will be the impact of this work? 


Recognizing that there are different levels of understanding 
about the work of LAMP, we have developed a primer that 
outlines the genesis and intent of the LAMP initiative. The LAMP Primer is 
available at: www.dhe.mo.gov/LAMP. 





LAMP Subcommittee Updates 


The LAMP Subcommittees have developed action plans 
and have been working to achieve identified goals. The 
Assessment Practices and Literature Review 
Subcommittees, in particular, have been hard at work in 
their respective groups. These efforts have been carried 
forth by core groups of dedicated participants. The 
information gathered and synthesized by the 
Subcommittees will serve as a foundation as the large group develops 
recommendations for the Commissioner by June 1, 2009. See below for updates 
on Subcommittee activities. 





Assessment Practices Subcommittee: 


The Assessment Practices Subcommittee has met several times over the last few 
months. In January 2009 they fielded a survey of assessment practices to all 
institutions in Missouri and have worked the past several months to code and 
analyze their results. 


The Assessment Practices Subcommittee is working to finalize their draft report 
which reflects upon the infrastructure and general role of assessment practices in 
Missouri institutions, as reported in the Survey of Assessment Culture distributed to 
institutions in January 2009. The survey also collected additional information from 
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institutions regarding their opinions and policy recommendations related to a 
breadth of relevant assessment issues. This information will also be used to inform 
the subcommittee's final report. 


Literature Review Subcommittee: 


ashley 


OP keeuulileseelos The Literature Review Subcommittee continues to work on finalizing the review of 
nd national best practices and has established an online database for review and 
annotation of assessment-related literature and a collaborative writing process for 
the production of their report. 


Communications Subcommittee: 


The Communication Subcommittee met on December 18", 2008. The group 
developed an action plan to keep both internal and external constituents informed 
and identified next steps; however, issues surrounding budgeting issues at the state 
E Facilitators level in early 2009 required that work with outside constituents be slowed to allow 
Prich focus on financial discussions. 


Recently, the group developed a “LAMP Primer” and is working to keep LAMP 
participants engaged and informed through avenues such as this newsletter. 


Policy Focus 


With much of the foundation for policy recommendations set forth, the next step 
will focus on development of recommendations to be delivered to the 
Commissioner. Given the current policy environment, Commissioner Stein has 
asked that while the group should continue to develop policy recommendations in 
all initial focus areas, it should give priority to recommendations regarding access 
and placement into postsecondary education. The focus placed here will allow the 
CBHE to develop informed policy as statewide discussions and policy development 
has already begun. LAMP will continue its work to provide substantive information 
to assist in policymaker decision-making in all areas. The letter from Commissioner 
Stein outlining this request to LAMP is available at: 





What’s Next? 


With the work of the subcommittees nearing completion, the next step is for the 
subcommittees to meet together as a large group to consider the information 
gathered and its impact on the final recommendations to the Commissioner. The 
May 11™ meeting will provide a forum for discussion and development of the June 
1* report. 
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Principles, Research, and Literature in 


Student Learning Assessment 


(A draft report by the LAMP Literature Review Subcommittee) 


Executive Summary 

The Literature Review Subcommittee reviewed the research and professional best practices as 
presented in the literature. Below is an outline of the major points as outlined in the 
subcommittee’s report. 


What do we know about fundamental principles of assessment? 


e Professionally accepted principles of assessment should guide assessment policy 
development. A summary of central principles include, but are not limited to: 


O 
O 


O 


Assessment is not an end in itself but a vehicle for educational improvement. 
Assessment should be based on multiple measures appropriate to the course, 
program, and institutional mission and goals. 

Assessment fosters wider improvement when representatives (e.g., faculty, 
administrators, assessment professionals) from across the educational community 
are involved. 

Assessment is most likely to lead to improvement when it is part of a larger set of 
conditions that promote change. 

Assessment should be an ongoing cumulative process. 

The data collected should be longitudinal and should include both quantitative 
and qualitative elements. 

Assessment programs should be based on reliable research and proven practices. 
Assessment instruments and methods should be continually evaluated to 
determine their utility in the assessment process. 

Assessment is linked to strategic planning and program review processes within 
the institution. 

Through assessment, educators meet responsibilities to students and to the public. 


e There are multiple purposes of assessment. They include: 


O 


O 
O 
O 


Improvement of student learning 

Improvement of program of instruction 

Improvement of educational effectiveness of the instruction 
Documenting student learning, program improvement, and educational 
effectiveness of outside stakeholders—Accountability 


e Assessment should be chosen to primarily serve continuous improvement in student 
learning, but with the ability to serve additional purposes, e.g., program improvement, 
accountability. 


O 


Assessment chosen primarily for the purpose of accountability does not 
necessarily support other types of assessment (e.g. student learning). 
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Assessment differs from evaluation, though assessment includes multiple acts of 
evaluation: 

o Evaluation is a judgment in relation to a goal or standard. 

o Assessment is a process of measuring performance and providing documentation 
of growth or feedback for improvement. 

There are multiple levels of assessment that concern different units of analysis: 

o Course-Level Student Learning Assessment: measurement of specific intended 
student learning outcomes from a course; can be formative (throughout the 
course) or summative (end-of-course). 

o Program-Level Student Learning Assessment: student learning outcomes upon 
completion of a program of study; can take place throughout a program or as end- 
of-program exams. 

co Institution-Level Student Learning Assessment: general competencies expected to 
be attained by some or all students by the end of their programs; the most 
common example is assessment of general education skills. 

In addition, there are also levels of performance that refer to the attainment of the ability 
identified as a learning outcome. 

o Levels of performance lie along a continuum of ability or achievement, and 
different levels of performance may be expected for the same learning outcome 
over time (e.g., a high school student may perform at an exceptional level on a 
particular learning outcome that would not be considered acceptable at the 
postsecondary level). 

o Levels of performance must be included in articulation of learning outcomes. 


What do we know about access and placement? 





While society is well on its way to the goal of universal access to postsecondary 
education, large numbers of high school graduates enter postsecondary education 
institutions unprepared for college-level study—federal estimates indicate 40% of 
students take at least one remedial course. 

The purpose of assessment to access and placement issues is to ensure effective 
placement decisions that increase academic success. 

While high school students who complete -eollege preparatory” curriculum are generally 
better prepared for college, far too many of these students need developmental/remedial 
education. 

Misalignment of course material, tests, and standards between high school and college 
remains a major challenge. 


A promising policy in other states has been the use of college placement exams as 
diagnostic tools to provide early feedback to high school students about progress toward 
college readiness. 
Assessment for placement into collegiate-level coursework involves diagnosis of a test 
score that is correlated with a reasonable chance of success in a particular course. 
o Oklahoma has seen success with this use of placement with a reduction of in 
remedial enrollments since the establishment of statewide minimum ACT —first 
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cut score” for access to collegiate-level coursework, with institutions given 
autonomy to place students who fall below state standards. 
e The Literature Review group came to the follow guidelines based on the above 
knowledge: 

o Focus on an essential set of standards that are most important for future success in 
college. The common expectation should be for all high school students to take a 
rigorous core curriculum, regardless of plans for college. 

o There must be clear performance expectations of college readiness so that 
students, parents, and educators receive consistent messages about what it means 
to be prepared for college. 

o Early monitoring and intervention with middle school students must occur to keep 
them on target and/or diagnose weak areas that must be addressed. 

o The ability to compare standardized measures across institutions and peer groups 
can lead to clearer standards and positive outcomes. 


Introduction 

The Learning Assessment in Missouri Postsecondary Education (LAMP) Advisory Council was 
created to consider statewide issues surrounding learning assessment in Missouri and to make 
policy recommendations to the Commissioner of Higher Education. The purpose of this 
literature review is to present a review of learning assessment research and best practices 
literature upon which the LAMP Advisory Council may base its recommendations to the 
Commissioner. Learning assessment research is highly contextual and few, if any, universals 
apply. To necessarily narrow the scope of this work, the LAMP Advisory Council chooses to 
focus on issues and policy questions that are important to the state of Missouri and were 
foundational to the creation of LAMP: access and placement (student preparation), beginning 
general education course transfer, and college-level general education. National and state-level 
concerns with remedial education require that LAMP first prioritize policy recommendations in 
access and placement. This report will examine the methodology used by the Literature Review 
Subcommittee of LAMP for conducting the literature analysis, provide a brief primer on the 
principles and purposes of assessment, provide an analysis of the literature as it relates to access 
and placement, and outline conclusions for increasing student success based upon the literature 


Methodology 

To perform this review of research and best practices, searches were conducted in three distinct 
areas of research publication and in the professional literature associated with each of the focus 
areas. The three distinct areas of research publication were (1) learning assessment policy 
research, (2) learning assessment research, and (3) learning research. 


Shavelson, writing about alternative designs for examining student outcomes from telecourses, 
observed that evaluators have a wide range of alternative designs from which to choose: "Which 
choice is best for a given situation depends on many factors, not the least of which are the types 
of decisions (and decision makers) on which the evaluation focuses and the feasibility of 
implementing the design." (Shavelson, R. et al, 1986, p. v) In a later section, an important 
distinction will be drawn between evaluation and assessment in the context of this review, but 
the observation applies just as much to the design of learning outcomes assessment as it does to 
program evaluation. 
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Learning Assessment Policy Research 

Research on learning assessment policy addresses the effectiveness of assessment policy in 
achieving its goals. An extensive literature review of learning assessment policy research was 
published by the National Center for Postsecondary Improvement (NCPI) in 1997. NCPI was a 
collaborative research partnership of Stanford University, the University of Pennsylvania, and 
the University of Michigan. While NCPI ceased operations in 2004, its research findings, 
publications, and toolkits continue to be available at http://www.stanford.edu/group/ncpi/, 
maintained by the Stanford Institute for Higher Education Research. In matters of learning 
assessment policy research, the related publications of NCPI have served as a base. Literature 
searches original to this review will be limited to the time period, 1997 to the present. Sources of 
learning assessment policy research mentioned in the NCPI "Benchmarking" report will be 
searched for new publications since 1997. These sources include federal agencies, state 
governments, regional accrediting associations, voluntary associations of colleges and 
universities, the National Governor's Association (NGA), State Higher Education Executive 
Officers (SHEEO), Education Commission of the States (ECS), and the National Center for 
Higher Education Management Systems (NCHEMS). 





Learning Assessment Research 

Research on learning assessment addresses the effectiveness of assessment strategies, techniques, 
and instruments in improving student learning, informing academic program improvement, and 
meeting accountability requirements. For example, when feedback is given, timely and 
actionable feedback improves learning much more than simple knowledge of results (Nyquist, 
2003). 


Learning Research 

Research on learning includes both basic and applied research on how people learn. In this 
review, the emphasis is on learning research that may inform learning assessment practices and 
policies. For example, researchers have found that testing enhances learning more than additional 
study of the material, even in the absence of feedback (Roediger & Karpicke, 2006). 


In addition to published research on learning assessment policy, learning assessment, and 
learning, this review covers professional literature on best practices within the focus areas. 
Beginning in 1989, the bimonthly publication, Assessment Update, has covered developments in 
higher education assessment. Other periodicals that regularly report on best practices in learning 
assessment in postsecondary education include Change, The National Teaching & Learning 
Forum, and AAC&U's Liberal Education and Peer Review. 


Assessment 101 

The term "assessment" has many meanings in ordinary language and in various technical 
languages. Assessment in this context means student learning outcomes assessment in 
postsecondary education. 


Many recent books and articles on assessment in higher education date the beginning of current 


concerns with assessment in higher education in the United States from the 1980s. Frequently 
cited as prompts are publication of A Nation at Risk (NCEE, 1983), Involvement in Learning 
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(NIE, 1984), Time for Results (NGA, 1986), and Boyer's College (1987). A good case can be 
made that these and other publications during this time stimulated a new national concern with 
assessment for accountability purposes, but assessment as a means of measuring and improving 
learning in higher education has a much longer history. By some accounts, assessment as a 
means of measuring learning was practiced as early as the 4th century, B.C.E., during the Han 
Dynasty in China (Biggs, J., 2001). However, the purpose of assessment then, and in 
contemporary times through the 1940s, was primarily to screen and select those most capable, or 
incapable, of learning. Informal assessment to improve learning is of course as at least as old as 
recorded accounts of teaching, made famous in Plato's accounts of Socrates. But the 
contemporary use of formal assessment to improve learning in higher education might be dated 
from the beginnings of the competency-based reform movement in higher education during the 
late 1960s and early 1970s (Grant, G. et al, 1979). 


Principles of Assessment 

This section on principles of assessment must begin by acknowledging and seeking to build upon 
the document, Guiding Principles of Assessment (GPA), developed by the Missouri Assessment 
Consortium (MAC) in 1992. The MAC statement of assessment philosophy opens with the 
following assertion: "Assessment should be guided by clearly stated, externally validated student 
learning processes and outcomes that flow from and support the institutional mission." In other 
words, assessment should be guided by what we know about how people learn and focused on 
learning objectives that flow from and support the mission of the institution in which assessment 
takes place. While this opening statement asserts that principles of assessment should be guided 
by principles of learning, the clear emphasis of the opening paragraphs of the MAC GPA is on 
preserving the autonomy of degree-granting postsecondary institutions. This emphasis is best 
understood in light of the historical context of the document's creation, a time in which pressure 
from federal and state government was building on public institutions of higher education to 
provide more evidence that students were learning what institutions said they should be learning 
and were learning. 


The MAC GPA identifies three purposes of assessment: "A) improvement of student learning 
and instruction, B) accomplishment of institutional mission, and C) accountability for 
achievement of educational goals." Irrespective of purpose, the following are identified as 
important features of assessment: 


e Assessment should be based on multiple measures appropriate to the program and 
institution 

e The data collected should be longitudinal and should include both quantitative and 
qualitative elements 

e Assessment programs should be based on reliable research and proven practices 

e assessment instruments and methods should be continually evaluated to determine 
their utility in the assessment process 


Several organizations have created lists of principles of assessment. Perhaps the most frequently 
cited in higher education are those published originally in 1996 by the American Association of 
Higher Education (AAHE). AAHE was dissolved in 2005 but AAHE's 9 principles of 
assessment can still be found on many assessment websites. The following abbreviated list is 


36|Page 


adapted from a more complete version retrieved from 
http://www. facet.iupui.edu/resources/AAHE%20Principles.pdf 


1. 


2. 


The assessment of student learning begins with educational values. Assessment is 
not an end in itself but a vehicle for educational improvement. 

Assessment is most effective when it reflects an understanding of learning as 
multidimensional, integrated, and revealed in performance over time. Learning is a 
complex process. It entails not only what students know but what they can do with 
what they know; it involves not only knowledge and abilities but values, attitudes, 
and habits of mind that affect both academic success and performance beyond the 
classroom. 

Assessment works best when the programs it seeks to improve have clear, explicitly 
stated purposes. Assessment is a goal-oriented process. It entails comparing 
educational performance with educational purposes and expectations — those 
derived from the institution's mission, from faculty intentions in program and 
course design, and from knowledge of students' own goals. 

Assessment requires attention to outcomes but also and equally to the experiences 
that lead to those outcomes. Information about outcomes is of high importance; 
where students "end up" matters greatly. But to improve outcomes, we need to 
know about student experience along the way — about the curricula, teaching, and 
kind of student effort that lead to particular outcomes. Assessment can help us 
understand which students learn best under what conditions; with such knowledge 
comes the capacity to improve the whole of their learning. 

Assessment works best when it is ongoing not episodic. Assessment is a process 
whose power is cumulative. Though isolated, "one-shot" assessment can be better 
than none, improvement is best fostered when assessment entails a linked series of 
activities undertaken over time. 

Assessment fosters wider improvement when representatives from across the 
educational community are involved. Student learning is a campus-wide 
responsibility, and assessment is a way of enacting that responsibility. 

Assessment makes a difference when it begins with issues of use and illuminates 
questions that people really care about. 

Assessment is most likely to lead to improvement when it is part of a larger set of 
conditions that promote change. 

Through assessment, educators meet responsibilities to students and to the public. 
There is a compelling public stake in education. As educators, we have a 
responsibility to the public stakeholders that support or depend on us to provide 
information about the ways in which our students meet goals and expectations. But 
that responsibility goes beyond the reporting of such information; our deeper 
obligation — to ourselves, our students, and society — is to improve. Those to 
whom educators are accountable have a corresponding obligation to support such 
attempts at improvement. 


(Authors of the AAHE Principles included Alexander W. Astin, Trudy W. Banta, K. Patricia Cross, Elaine El- 
Khawas, Peter T. Ewell, Pat Hutchings, Theodore J. Marchese, Kay M. McClenney, Marcia Mentkowski, Margaret 
A. Miller, E. Thomas Moran, and Barbara D. Wright.) 
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A third set of principles often cited are those published as the National Association of State 
University and Land Grant Colleges' (NASULGC) "Statement of Principles on Student 
Outcomes Assessment" Interestingly, these principles are not posted on the NASULGC website. 
The NASULGC principles state that programs for student outcomes assessment should: 


1. focus primarily on the effectiveness of academic programs and the improvement of 
student learning and performance; 

be developed in collaboration with the faculty; 

be appropriate to the particular mission and goals of the institution; 

use multiple methods of assessment; 

be fiscally conservative and not impose costly programs on institutions; 

be linked to strategic planning and program review processes within the institution. 


Oy ve tS 


The published lists of assessment principles above focus primarily on program-level and 
institution-level assessment, and assume a high level of knowledge of, and experience with, the 
terms of discourse and literature on assessment in higher education. The following "Assessment 
101" section may help those who have not participated in that discourse or read extensively in 
that literature. 


Purposes of Assessment 
The appropriateness of any method of assessment or assessment instrument depends on the 
purpose of assessment. The following purposes of assessment are considered in this review: 


Improve Student Learning 

Improve Program of Instruction 

Improve Educational Effectiveness of the Institution 

Document Student Learning, Program Improvement, and Educational Effectiveness 
to Outside Stakeholders (Accountability) 


oS 


Over the past twenty years, assessment for the purpose of accountability has become a 
dominating concern in higher education. The problem, many observers now agree, is that 
methods of assessment and assessment instruments developed or chosen solely or primarily for 
purposes of accountability do not necessarily serve to improve student learning, improve 
programs of instruction, or improve educational effectiveness at the institutional level. The 
challenge is to develop or choose methods of assessment and assessment instruments primarily 
for the purpose of improving student learning that can also serve purposes of program 
improvement, educational effectiveness of the institution, and accountability to external 
stakeholders. 


Differentiating Assessment from Evaluation 

In many contexts, no distinction is made between the meanings of assessment and evaluation. In 
this context, it is important to distinguish assessment from evaluation. Assessment is a process of 
measuring a performance or product of learning and giving feedback which documents growth 
and provides directives to improve the performance or product. Evaluation is a judgment or 
determination of the quality of a performance or product in relation to a goal or standard. Some 
efforts to distinguish assessment from evaluation attempt to define them in ways that make them 
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seem mutually exclusive (e.g., Parker, P. et al, 2001). Some efforts force the meanings of 
assessment and evaluation apart by equating the former with formative evaluation and the latter 
with summative evaluation as first distinguished by Michael Scriven (Scriven, M., 1967). In this 
context, it would be most accurate to say that assessment includes multiple acts of evaluation, 
but is more than evaluation. Documentation of growth and actionable feedback to improve 
learning are as essential to assessment as is evaluation. 


Assessment for/as Learning versus Assessment of Learning 


Distinguishing assessment "for" learning or assessment "as" learning from assessment "of" 
learning is perhaps not necessary if the previous differentiation of assessment from evaluation is 
already recognized and accepted. Unfortunately, in practice, assessment is not routinely 
differentiated from evaluation and assessment "of" learning is taken to mean the same thing as 
summative evaluation, a judgment of a performance or product at the conclusion of a learning 
experience. This has led to the development of the distinction in assessment literature between 
assessment "for/as" learning and assessment "of" learning, with assessment "for/as" learning 
intended to mean the formative process that here we equate with assessment. But the phrasing of 
assessment "for" learning and assessment "as" learning can still contribute extra meaning even 
when it is recognized and accepted that all assessment is formative by definition. The valuable 
extra meaning supplied by using the prepositions "for" or "as" is the intention that the assessed 
demonstration of learning is itself a learning experience, or that the complete process of 
assessment-performance, evaluation, documentation, feedback-be as brief and tightly connected 
as possible. Assessment of a "real-world" performance or performance in a high fidelity 
simulation of a "real-world" setting would be an example of assessment for learning. Learning to 
lengthen or deepen a meditative state using biofeedback equipment, would be an example of a 
very brief and tight performance-evaluation-documentation-feedback loop. 


Levels of Assessment and Levels of Performance 

Levels of Assessment. It is also important to identify and distinguish levels of assessment and 
levels of performance because the term "level" is used in both cases but means something very 
different. By levels of assessment, we are referring to course, program, and institutional, levels 
of student learning assessment data collection or data analysis. 


1. Course-Level Student Learning Assessment 

All courses have, or should have, specific intended student learning outcomes. For example, 
students in an Introduction to Macroeconomics course need to be able to calculate real GDP. The 
assessment of course-level learning outcomes can take place throughout the course and can be 
measured through a very wide variety of typically faculty-based tools such as quizzes, tests, 
papers, portfolios, journals and class assignments or other artifacts. Formative course-level 
assessment requires multiple in-course assessments to improve student learning. End-of-course 
assessments, such as a final exam, or final paper or project evaluation, are summative with 
respect to the individual student's learning in that specific course, but can be formative if part of 
a sequence of courses in which the student's learning in later courses can be improved based on 
the feedback received in a previous end-of-course assessment. 
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2. Program-Level Student Learning Assessment 

All degree programs in postsecondary education have, or should have intended program-level 
student learning outcomes. For example, students in a Bachelor of Science in Nursing program 
should, by the time they graduate from the program, be able to explain and implement triage to a 
patient. The assessment of program-level learning outcomes can take place throughout a 
student's program in more than one course. End-of-program exams are sometimes also referred 
to as learning assessments, but the value of such exams is obviously limited to program 
improvement. In other words the end-of-program exam, just like end-of-course exams, are 
summative with respect to the particular student tested; they are potentially formative only with 
respect to improvement of the academic program. When program competencies are tracked 
throughout a student's coursework, the college typically has a paper or electronic tracking system 
to insure sufficient success on program competencies. Program-level learning outcomes can also 
be assessed at the end of the program. And end-of-program exam may be locally developed by 
program faculty or it may be a standardized exam given to students graduating from similar 
programs across the state or nation. 


3. Institution-Level Student Learning Assessment 

The most common examples of institution-level student learning assessment are assessments of 
the general education program required of undergraduate students across many programs, and 
proficiency assessments of general competencies expected to be attained by some or all students 
across many programs by the end of their program. For an example of the first type, at or near 
the time that a student completes all or most of her or his general education requirements for a 
two-year or four-year degree program, the student might be required to take one or more general 
education assessments, such as a writing assessment and an assessment of critical thinking or 
broad content knowledge, that target intended learning outcomes of the general education 
program. Such assessments could be formative with respect to the student, if the student gets 
feedback that the required level of performance has not been achieved and there are opportunities 
for the student to improve. Even if summative for the student, such a general education 
assessment can be formative for the institution if the results are used to continuously improve the 
general education program. Examples of the second type, assessments that are taken by students 
across many programs at the very end of their academic programs may be similar to general 
education assessments but calibrated to higher levels of performance, or they may be 
substantively different than assessments given to assess outcomes in the general education 
program, such as integration of multiple competencies in a field of specialization. 

Some colleges refer to their college-wide, end-of-program learning objectives as "Common 
Student Learning Outcomes" or as "Common Student Abilities", etc. 


Some regard institutional performance indicators such as retention rates and graduation rates, as 
part of institution-level assessment, but such indicators are not assessments of learning. Our 
review is limited to assessment of student learning. 


Levels of Performance. By levels of performance we are referring the level of attainment of the 
ability identified as a learning outcome. For example, it is very common to see competencies in 
communication and critical thinking identified as key learning outcomes at different levels of 
education from high school to graduate school. But for any general competency, there is a 
continuum of ability or achievement and we do not expect the same level of performance in high 
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school that we expect in earning an associates degree, a baccalaureate degree, a master's degree, 
or a doctoral degree. There is of course overlap in the levels of performance a student may 
demonstrate. A high school student may perform at a level that is exceptional in terms of our 
expectations for high school and that would be adequate if not exceptional at a collegiate level. 


The two important points here are: (1) to understand the different meanings of levels of 
assessment and levels of performance, and (2) to understand that levels of performance must be 
included in the articulation of learning outcomes at all levels of assessment and levels of 
education before appropriate assessments can be developed or chosen. 


Assessment Related to Access and Placement 

While it might seem that, as a society, we are well on our way to achieving the goal of universal 
access to postsecondary education, large numbers of high school graduates enter postsecondary 
education institutions unprepared for college-level study (Greene & Foster, 2003). David Conley 
defines college readiness as —th¢evel of preparation a student needs in order to enroll and 
succeed—without remediation—in a credit-bearing general education course at a postsecondary 
institution that offers a baccalaureate program or transfer to a baccalaureate program” (Conley, 
2007, p. 5). However, federal estimates indicate that 40% of admitted and enrolled students take 
at least one remedial course (National Center for Education Statistics, 2004). Even ten years ago, 
according to Breneman & Haarlow (1997), the costs of remediation were estimated at $1 billion 
or more at public institutions alone. It would seem a "no brainer" that states would look for ways 
to reduce the need for remediation, but while many states have assessment policies governing 
assessment of college readiness at entry (at least in English and math) and placement, few have 
policies in place to address the problem of preparation. 


The purpose for assessment, as it relates to access and placement of students into collegiate-level 
or pre-collegiate level coursework, is to assist institutional personnel in making course placement 
decisions that will help students become academically successful. Access to collegiate-level 
coursework refers to assessing the basic skills of incoming students. Placement refers to the 
enrollment of students into collegiate-level coursework (credit-bearing coursework toward 
degree attainment), or pre-collegiate level coursework (remedial or development courses that are 
often non-credit bearing and do not count toward degree options) if the student is unable to 
demonstrate a certain level of proficiency. 


It is well documented that high school students who complete a so-called college preparatory 
curriculum are generally better prepared for college than those who do not (Conley, 2007). But 
far too many students who do complete a college preparatory curriculum are still found to need 
remediation courses once they enter college (ACT, 2007 National Data Release). A study 
conducted by the Ohio Board of Regents in 2002 found that 25 percent of Ohio high school 
graduates with a known core curriculum required remediation in math or English (Long & Riley, 
2007). Even higher percentages of presumably well-prepared California high school graduates 
have been found to require remediation in math and/or English upon entry at California State 
University and University of California campuses (Long & Riley, 2007). 


41|Page 


College Readiness and the Misalignment of Standards 

The problem runs deeper than just poor preparation in high school. The deeper problem has been 
identified as a misalignment of course material, tests, and standards between high school and 
college (McCabe, 2001; Venezia, Kirst & Antonio, 2003; Conley, 2007). Aligning curriculum 
between secondary and postsecondary levels requires a sustained and coordinated effort. 
Aligning curriculum graduates better prepared students who experience more successful 
transitions from high school to college and helps to streamline education (Achieve, 2008) 


Missouri’s alignment process, the Curriculum Alignment Initiative, began in 2007 and is driven 
by recommendations of the P-20 Council, the Missouri Math, Engineering, Technology, and 
Science Coalition, and by the legislatively-driven mandates of Senate Bill 389. CAI established 
competencies for entry- level and exit-level coursework. The goal of the entry-level 
competencies is to set a clear standard for students, parents, legislators, and educators of what 
skills students need to be successful in college. The entry competencies set a minimum threshold 
that students must meet in order to gain access to collegiate-level coursework. Senate Bill 389 
further mandates that the competencies be provided to the Missouri Department of Elementary 
and Secondary Education for their review and for them to align their assessments with the 
competencies. Entry-level competencies have been developed for the following disciplines: Arts 
and Humanities, Mathematics, Science, Social Science, English and Communication, Foreign 
Language, and Cross-Disciplinary Skills. Research is clear that cognitive and academic 
behaviors are beneficial to student preparation, but the -kabits of mind” and contextual, and 
personal behaviors skills and abilities are also crucial for student success (Conley, 2007). Exit- 
level competencies were also established through CAI, but their primary purpose is aiding in 
transfer and articulation of single course transfers outside the 42 hour block. Exit competencies 
will be discussed in more detail in future work outlining policy issues related to transfer and 
articulation. 


LAMP is considered the natural next step after the creation of the competencies. One may well 
ask how we know whether students exhibit these competencies if they are not measured. One 
function for LAMP is to -turn the spotlight on assessment” and utilize the competencies as a 
guide for determining student preparedness for collegiate-level coursework. Questions that arise 
for consideration include: 


e Are their essential entry competencies important to access and college readiness 
that have yet to be addressed by the Curriculum Alignment Initiative? 

e Howcan we best assess CAI Entry Level Competencies prior to postsecondary 
entry to alleviate the need for remedial/developmental/pre-collegiate coursework at 
or after entry? 

e What still needs to be done to align CAI Entry Level Competencies with DESE 
educational assessment standards like Course Level Expectations (CLE)? 

e Incases where CLEs are adequately aligned with entry-level competencies, are the 
End-of-Course examinations (EOC) of the CLEs sufficient to assess for access to 
postsecondary coursework? 

e What kinds of supplemental assessment are required if EOC's are not sufficient 
and/or for exceptions like late transfer students, out-of-state students, advancement 
from remediation/developmental coursework to college level etc.? 
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e How do we ensure that Dual Credit students meet the same expectations as other 
students? 

e How can we best assess the entry-level competencies at postsecondary entry to 
most effectively address needs for remedial/developmental/pre-collegiate 
coursework at or after entry? 


Early Diagnosis 

The use of college placement exams as diagnostic tools in high school is one promising policy 

that has been pursued in several states. (Tierney & Garcia, 2008). Ten states are administering 

college and career readiness tests to all high school students as a result of statewide assessment 
systems (Achieve, 2009) 


Clear Standards for Placement 

The Missouri Assessment Consortium (MAC), in creating the MAC Handbook, sought to fulfill 
the need for a reference resource of assessment practices and experiences at public four-year 
institutions in Missouri. The Handbook also provides definitions of key terms used in assessment 
that this paper will utilize in providing a basis for research. 


Placement and Diagnosis. In the context of assessment for placement into collegiate-level 
coursework, diagnosis has been defined, according to MAC, as —thmeaningful association of a 
test score with a local education experience. In other words, after careful study the institution has 
determined that students earning a score below a given point will not be successful in a particular 
course or pattern of courses without institutional intervention and individual scheduling 
decisions.” Placement into remedial/development/pre-collegiate level coursework occurs when 
the institution establishes —eut scores’ for placement in enrichment or remedial/developmental 
sections.” 


In 1994, the Oklahoma State System for Higher Education adopted several initiatives in their 
efforts to reduce remediation including: enhancing teacher preparation; increasing standards for 
college preparation; establishing better communication and feedback to Oklahoma high schools; 
initiating programs to enhance cooperation between state institutions; and improving Oklahoma 
college and university graduation rates. In 1994, the Oklahoma State Board of Regents adopted 
the Student Assessment Policy requiring each institution to develop and implement a 
comprehensive assessment program with mandatory student placement. Institutions are required 
to administer a standard comprehensive assessment tool, in this case, the ACT, and to use an 
ACT score of 19 as their "first-cut" in the areas of English, Math, Science Reasoning, and 
Reading. Scores below 19 require students to enroll in remedial courses or undergo secondary 
assessments. Although all institutions use the ACT as the first entry-level assessment, secondary 
evaluation testing instruments vary according to the institution. Most institutions use ASSET, 
AccuPlacer, COMPASS, and/or the Nelson-Denney Reading Test, and each institution is 
responsible for establishing their own cut-scores. These pre-collegiate level courses do not count 
toward degree requirements and a supplementary per credit hour fee is assessed the student for 
these courses. Colleges offer orientation courses, computer-assisted instruction, tutoring, and 
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learning centers, in an effort to increase the rate at which students who take pre-college level 
courses succeed. Institutions are required to report to the Oklahoma State Regents the methods, 
instruments, and cut-scores used for entry-level course placement, as well as the student success 
in both remedial and college-level courses. High school students wishing to concurrently enroll 
in courses with established ACT cut-scores will not be allowed to enroll in those courses if they 
score below the minimum standard. A student who scores below the established ACT score in 
reading is not permitted enrollment in any other collegiate course. Secondary institutional 
assessments and remediation are not allowed for concurrent high school students. 


Results show that since the inception of Oklahoma’s assessment policy, the percent of first-time 
freshmen enrolled in remedial courses has decreased in the state system. From 1996-97 to 2006- 
2007, the percentage of first-time freshmen enrolled in remedial courses decreased from 40.3% 
to 36.5%. At research institutions, the percentage dropped from 21.3% to 6.7%, and at regional 
institutions, the percentage dropped from 34.0% to 33.0%. At community colleges, the 
percentages remained the same at 49.9%. 


Conclusions 

The literature points draws a number of conclusions for increasing student success and reducing 
the number of students who take pre-collegiate level coursework as a result of their being 
underprepared for college. 


e Focus on an essential set of standards that are most important for future success in 
college. The common expectation should be for all high school students to take a 
rigorous core curriculum, regardless of plans for college (ACT, 2008; Conley, 
2007). 

e There must be clear performance expectations of college readiness so that 
students, parents, and educators receive consistent messages about what it means 
to be prepared for college (ACT, 2008; Conley, 2007). 

e Early monitoring and intervention with middle school students must occur to keep 
them on target and/or diagnose weak areas that must be addressed (ACT, 2008; 
Achieve, 2009;Tierney & Garcia, 2008). 

e The ability to compare standardized measures across institutions and peer groups 
can lead to clearer standards and positive outcomes (Dwyer, Miller, & Payne, 
2006; OSU, 2008). 
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LAMP Discussion Themes 


The LAMP Advisory Council reconvened as a whole body in May 2009 with a purpose to 
engage participants in focused discussions on the primary questions, issues, and potential options 
for policy makers. In a May 7", 2009 letter to LAMP participants, Commissioner Stein 
encouraged the Advisory Council to build upon the research conducted by the subcommittees 
and focus their efforts on policy issues related to promoting greater student preparation and 
success. Three levels for policy recommendations were identified: 


1. Determine where consensus exists among the group and make clear policy 
recommendations, 

2 Determine where further research and examination is warranted and outline a 
strategy for progress, and 

2; Describe issue areas where there remains significant disagreement or lack of 


clarity and controversy. 


Although the meeting participants were able to determine areas where consensus exists among 
the group, the statements do not rise to the level of policy recommendations. Areas where the 
group reached consensus were: 


I: The Literature focuses on math, reading, writing, and critical thinking skills as the 
most important for student success in collegiate-level coursework. 
2 Postsecondary and secondary collaboration, with routine feedback is imperative 


for student success. 

Multiple-assessment model approaches to student measurement are vital. 

4. The literature supports early-assessment models for measuring added value in 
learning. 


ios) 


The group decided that further research was needed in regard to using the Curriculum Alignment 
Initiative competencies to map student success. CAI worked on developing outcome statements 
for what students should know, but more research is necessary to determine whether the 
competencies are aligned with secondary curriculum and current assessment tools. 


The next step for the LAMP Advisory Council is to identify potential policy recommendations 
drawn from the conclusions and to outline a strategy for obtaining information about the 
competencies, through pilot projects or alignment studies. 


45|Page 


Missouri Department of Higher Education 
Building Missouri's future...by degrees 
May 7, 2009 


Colleagues: 


First, let me take this opportunity to express my thanks and gratitude for your dedication and 
good work over these last months. The efforts of LAMP - a voluntary group of educational 
professionals - have resulted in providing an important foundation about the scope and 
magnitude of student learning assessment at Missouri’s colleges and universities, evidence of 
promising practices and theoretical frameworks from an extensive literature review, and 
principles for effective information dissemination for use by policymakers and practitioners. 
This work is invaluable as we collectively seek to fully implement the MDHE’s Curriculum 
Alignment Initiative (CAI) and the Coordinating Board’s public agenda for higher education, 
Imperatives for Change. 


As many of you know, the past few months have been tumultuous as the whole country faces 
dire questions about our future economic prosperity and higher education focuses on how to 
secure adequate funding during this recession. It is commendable that we continue to move 
forward on the LAMP initiative even as this storm is raging around us. 


Although the dust surrounding funding issues for the immediate future is beginning to settle, 
elected officials are continuing to place a spotlight on the importance of teaching and learning at 
Missouri’s educational institutions. At local, state and national levels there is a sense of urgency. 


LAMP was created to provide policymakers with access to factual information and evidence of 
best practices that will positively impact policy development surrounding the assessment of 
student learning. While all transitions along P-20 educational pathways and into the workforce 
are important, it is essential to prioritize your work. The impressive state-level competency work 
completed to date will be for naught, unless, assessment policy follows that will impact 
assessment practices. 


Clearly all of the transitions along the P-20 highway — preschool to kindergarten, elementary to 
middle school, middle to high school, high school to college, lower to upper division work, and 
undergraduate to graduate education — are important. At the same time it will be more effective 
to focus on one area at a time. Therefore, I am charging you to prioritize your work by focusing 
first on the transition from secondary to postsecondary education. 


The need to ensure that more of our citizens attend and are successful in completing 
postsecondary educational programs is greater than ever before in our history. Public policy 
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about access to and placement in collegiate-level coursework are two areas that have great 
potential to effect major change in the preparation of students in the P-12 pipeline. 


I look forward to receiving an initial report and set of recommendations by June 1, 2009 about 
public policy surrounding assessment associated with access to and placement in collegiate level 
coursework. Thank you again for your continued efforts and commitment to this important 
work. 


Sincerely, 


a 
Y \, ——p — 
Kolsci| D—— — 





Dr. Robert B. Stein, Commissioner 
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LAMP Meeting 
May 11, 2009 
Large Group Discussion Themes 


CAI Entry-level Competencies and Assessment 
CONCLUSION: The literature almost exclusively focuses on math, reading, 


writing, and critical thinking skills as most important for student 
success in collegiate-level coursework. 


Should we attend to all CAI competencies or limit to a subset of CAI competencies as 
listed above as our focus in access and placement? 

Attempts to address all of the entry-level competencies may result in ineffective policy— 
simply too broad. Need to focus attention. -d rather do a few things well than a lot of 
things poorly.” 

The focus on a subset of competencies seems to be leading away from implementation of 
the identified entry-level competencies. 

Pilot projects can address assessment tools and whether they are aligned with and 
accurately assess established competencies. 

Look at secondary EOC exams to provide feedback re: alignment. 


Collaboration with Secondary Partners 
CONCLUSION: Best practices in literature supports collaboration with secondary 


educators. 
Institutions should collaborate with secondary feeder schools to provide more detailed 
feedback regarding the success of their students. 
Institutional faculty should establish partnerships with discipline-specific secondary 
educators to provide specific feedback regarding what students need for college (e.g. 
English faculty connect with secondary English teachers regarding what constitutes a 
good paper. 
Collaboration with secondary currently exists through CAI...why not use that process? 
Building Bridges Project—Northwest MO institutions collaborate with feeder schools to 
share data and look at whole picture of why student is successful or not successful. Using 
CAI competencies in math, English, and social studies to align curriculum. 


Standardized Cut-Scores 
CONCLUSION: Benchmarks and standards for demonstrating proficiency could suggest, 


not one standard, but standards that can be accepted across institutions. 
Oklahoma provides a range of scores, depending on the instrument used, as determined 
by the institution. 
We can establish scores autonomously by institution rather than an across-the-board 
establishment of standards (numbers). 


Assessment Standards 
CONCLUSION: Best practices in literature supports assessing student learning with a 


portfolio approach. 
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e Need to utilize a multiple-assessment model approach. Looking at scores alone only 
provides information in one area. Does not account for poor test-takers, unfocused 
students, —blow df students, etc. 


e Formative assessments provide best information. 


Points of Clarification/Areas of Further Research 
e Need to delineate between college success, admission to institutions, and course success. 
e CAI worked on outcome statements, but unsure whether competencies map to course 
success and success for moving on to the next course. 
e What are we defining as success? 
e Need further research to ensure validity of CAI competencies. 
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